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ABSTRACT 

To determine if there is a distinction between the 
forensics community's idea of quality and that of the general 
popula^^on, tournament rankings of forensics judges and those of a 
lay audience were compared. Undergraduate students enrolled in a 
variety of speech rv^lated courses were asked to attend rounds of 
competition at a midwest collegiate forensics tournament and fill out 
a ranking sheet (similar to a master ballot used by the regular 
assigned judges) on the performers. The comparative rankings 
generated 921 ranked pairs. Results showed that there does not appear 
to be a significant similarity in the direct rankings or adjacent 
rankings of professional and lay judges. The data also implies that 
even within the ranks of forensics judges a wide range of ranks is 
common and a wide spectrum of evaluation is likely. (Eight tables of 
data are included.) (MS) 
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When xcatching an 01>-mpic figure skater or a g.Mimast perform there is often a 
sense of au-e concerning the skills boang demonstrated. \<hon results are announced it 
isn't uncommon for audiences to wonder how a judge can distinguish among the perfor- 
mances. Television commentators will offer insight into the nuances of a performance 
pinpointing technicalities which malie or break a routine. The average audience mem- 
ber may mar%-el at the pickiness, or the highly technical criteria; reactions such as 
"I liked her the best, regardless of xchat the judges say" arc not uncommon. Foren- 
sics competitors perform in a similar ei.vironment . There an? spo<-ific writtr>n tulns 
which govern an event, such a^ time limits, and there are luiwritten conventions ^-iiich 
have become criteria for e%alual.ion, such as the tyj« of script Ixjoks for interpre- 
tive events. In addition there are regional attitudes which become critical factors 
in judging, such as the degree of movement or acting in a duo. All of these elements 
have developed into a highly specialized set of rules which govern the preparation, 
perfomiance and evaluation/ judging of a forensics uvenX.. Ojio jxji^sible outcome of 
this is a situation where Ihc forc-nsics communit>'s ideas of qu;»lit> become dramati- 
cally different from the general population. It seems that it would be useful to 
attempt to determine if such a distinction e>:ists. One jxissiblf way to studj- thi- ie: 
to c(«iipare the tuurnament rarJ-.ings of fon^nsic-s judges to i.hos..- of a lay aiidienc-. 
Two h>TX)thesis were suggested to test the eonditioii. 

H 1 "There is a significant similarity in the direc t tarjangs of forensics ijer- 
formances by forensics judges and ]ay judges." 

H 2 "There is a significant similaiity in the adjacent rankings of forensics 
performances by forensic? judges and la> judges." 

Two tei-rns may require clarification. A "direct" ranldng is a \xAv uf identical 
ranldngs, e.g. each judge ranks a contestant first. An "adjacent" ranking exists 
when judges ranlt wit.hin one pln>^o of each .^thor, e.g. one judge ranks a contestant 
second and the othc, judge rar.K- that contestant first or third. The rationale for 
considering th<' second hy|X)t.h. s;s ui 11 U- explaineii in th.; lesults s^e^tion. 
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METH0DOL0G\' 

I'ndergradtiate students enrolled m a variety of speech related courses were aske<i 
to attend rounds of competition at a midu*est collegiate forensics tournament and fill 
out a ranking sheet on the performers* This sheet w'as similar to a master ballot 
used by the regular assigned judges. The student/lay rankers were rt^ndomly assignexi 
to events and sections. In some instances more than one lay judge ranked the same 
set of performers. The forensicR judges and lay judges were instructed not to colla- 
borate with each other. 

For purposes of this initiaJ study the compai-ativc ranl\ings were i:<>inpiled for two 
intorprotivo events, (po(»tr\ ant. dnuiun interpretation) nrv^ twc> original events, 
(infonnative speal^ing and pt:rsuasiv(? spoalting). This gonorated 921 i-anl;cd pairs 
including 114 informative ranltings, 270 persuasive ranldngs, 216 drama interpretation 
rankings and 321 poetiy intorprc^talion ranlungs. 

The paired rankings for each ev(»nt wore subjected to rhi-squarc cros-> tcibula- 
tions. An additional cross tabulation was g(^nerated combining the ranl^ed pairs for 
all the events. This procedure provided data concerning the frequency of direct 
agreement between the forensics judge and the lay judge. 

In order to test the second hypothesis the ranlts were analyzed by comparing the 
frequency of adjacent ranliings using the following table: 



Results 

The results of the study related to the first h>7X)thesis arc summarized in tables 
2-6. The data gathered supports the null hMX)thesis; there does not appear to be a 
significant similarity in the direct racings of forensics and lay judges. The 
result of the clustering of adjacent rankings is summarize^] in tables 7-8. This data 



TABLE 1 



Forensics Judge Ranl<^ 



Lay Judge Adjacent Ranli 



1st 
2nd 
3rri 
•1th 
5th 



1st and 2nd 

1st, 2nd, 3rd 

2nd, 3rd, 1th 

3rd, Ith, 5th 

4th, 5th, 6th 



suggests that some similarity exists but the significance of this similarity is open 
to interpretation. The specific results and statistical analysis arc els follo^^s: 
Cross tabulation of ranlangs of forensics judge by lay judge 

TA BLE 2 
ALL El'ENTS 
Lay Judges 



Rank 



Forensics 1 11 .2%* 22-9% 15.3% 13.5% 7,1% 
Judge 40.2% 27.8% 15.2%. 13.9% 5.0% 



20.5% 29.8% n.9% 18,7% 11.1% 
20 . 1% 29 . 8% 19,9% 19 . 1% 7 • 9% 



18.9% 20.1% 26.0% 13.6% 21.3% 
18. 1% 19.9% 25.7% 13,9% 15.0% 



12. !% 15.3% 19. -1% 32. 1% 20.6% 
12.!% 15.2% 19.3% :*3,:3% 11.6% 



6.6% 8.7% 1-1.1% 13.3% 57.3% 
9.2% 12.3% 19.9% 19. 1% 57.5% 



♦The top number is the row percentage and the second number is the column percentage. 

With a five by five matrix there are sixteen degrees of freedom resulting in a 
critical value of 39.252 which would result in a probability level of .001. The 
chi-square x'alue for the comparison of all ranJ^s is 261.98 \;hich is t.'cll beyond the 
.0000 level of significance. noted at the beginning of this section this strongly 
supports the null hypothesis indicating that the lay judge and the forensics judge do 
not directly agree on the compjirativc value of a forensics performance. In looking 
at the raw percentages it can Ix? noted that ranlcings agree 50% of the time only uiien 
corujidering the 5th place ranlang. All judges were instruel(*d to ranl\ no lower than 
5th regardless of the number of contestants in a round \Aich meant that more than one 
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competitor could be i-anlted 5th thus increasing the likelihood of agreement among 
judges at this ranlc. It is interesting to note, however, that the next highest per- 
centage of agreement occurred in the 1st place ranldngs. This may support the notion 
held by some that it is easy to identify the top and bottom of a round but the middle 
three ranJts represent a grey area ;s+jere soocific distinction of rani? is difficult. 

The chi-square comparisons for each of the events tends to follou- the pattern of 
the combined data. There were some peculiar! tie? u-hich arc- noted following efich 
table. 

T.\BLE 3 

Cross tribulation of ranliinfes of forensics judge 
by lay judge 
INFORM.\TI\'E SPE-MvIKG 

Lay Judge 



RanJc__ 1 _? 3.. . 1. . .._..5... . 

Forensics ' 1 38.1%* 28.6% 14.3% 9.5% 9.5% 
Judge 38.1% 27.3% 15.0% 9.5% 6.7% 



28.6% 38.1% 9.5% l.Q^i 19.0% 
28.6% 36.4% 10.0% 1.8% 13.3% 



4.8% 14.3% 38.1% 23.8% 19.0% 
4.8% 13.6% 40.0% 23.8% 13.3% 



14.3% 9.5% 9.5% 38.1% 28.6% 
14.3% 9.1% 10.0% 38.1% 20.0% 



10.0% 16.7% 16.7% 46.7% 
13.6% 25.0% 23.8% 46.7% 



♦The top number is the row percentage, bottom number is the column percentage. 



The chi-square for thip event was 38.91 which reflected is a probability level of 
.0011. The anal.N-sis of this, event should be tempered by an understanding that the 
smaller sample size (114) resulted in 1 1 cells with an e\ent frequency of less than 
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5 10.0% 
14.3% 



five which does affect the final si^jnificanco assessment. Thr- assessment of raw per- 
centages again revealed that in all case? there icar less thai! 50% direct agreement 
for any given ranlcing. Also of interest is the consistenc> uf ranking agreement in 
the 1st through 4th place ranl^s. 

TABLE 4 

Cros?. tabulation of ranltings of 
forensics judge by lay judge 
PERSU/\SI\7i SPEAKINCi 

Lay Judge 



Rank. 1 2 3 !_ J 

Forensios 1 26.5% 10.2% 0 

Judge ^2.9% 27.7% 10.0% 8.3% 0 



8.2% 22.4% 36.7% 20. 1% 12.2% 
7 . 8% 2^ . 4% 36 . 0% 20 ,8% 8 . 1% 



16.3% 30.6% 20.4% 14.3% 18,4% 
15.7% 31.9% 20.0% 14,6% IL.2% 



20. 1% 6.1% 20.4% 30.6% 22. 1% 
19.6% 6.1% 20.0s: 31.3% 14.9% 



2.7% 6.8% 9.5% 16.2% 61.9% 
3,9% 10.6% 14.0% 2^.0% 64.9% 



♦Top number is the rot: percentage, bottom number is the column ]xjrcentage. 

The chi-squarc for persuasive speaking rtuil^s was 142.98 uiiich also was signifi- 
cant beyond the .0000 level. These results seem to most dramatically support the 
notion of ability to distinguish top and bottom rciJ^s with 1st place rank agreements 
of 55% and 5th place ranlt agr<?oments of 65% It is also interesting to note that this 
is the only instance when- no lay judges felt a forensics judge's 1st place was the 
worst performer in the rouiid. 
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TABLE 5 

Cross; tabulations of ranltings of 
I'oronsics judges by lay judges 
DRAMA INTERP 

I^y Judge 



Rani; 1 2 3_. _ }_ _ 5 

Forensics 1 48.7%» 20.5% 15.4%" 10.3% 5.1% 
Judge -18.7% 20.0% 15.4% 10.5% 3.3% 



2 25.0% 45.0% 5.0% 12.5% 12 5% 
25.0% 45.0% 5.1% 13.2^; 8.3% 



13.2% 13.2% 36.8% 18.4% 18.4% 
12.8% 12.5% 35.9% 18.1% 11.7% 



7.7% 10.3% 15.1% 35.9% 30.8% 
7.7% 10.0% 15.4% 36.8% 20.0% 



3.3% 8.3% 18.3% 13.3% 56.7% 
5.1% 12.5% 28.2% 21.1% 56.7% 



♦The top number is the row percentage, bottom is the column percentage. 

The chi-square for drama interp v-as 103.29 with a significant level of .0000. 
This event appeared to have the highest, incidence of dii*ect agreement at each rank 
with a range of 36 to 57% direct asrccmcnt . 
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TABLE 6 

Cross tabulation of ranliings of 
forensics Judges by lay Judges 
POETRY INTERP 

Lay Judge 



Rank 1 2 3 1 5 

Forensics 1 26. 2%* 19.7% 19.7% 21.3% 13.1% 
Judge 25.4% 19.4% 19.1% 22.4% 10.5% 



21.6% 23.0% 19.7% 2G.2% 6.6% 
23.0% 22.6% 19.4% 27.6% 5.3% 



29 .5% 18.0% 19 . 7% 6 . 6% 26 . 2% 
28.6% 17. 7X 19.4% 6.9% 21.1% 



8.2% 27.9% 24.6% 29.5% 9.8% 
7.9% 27. 1% 24.2% 31.0% 7.9% 



11.7% 10.4% 14.3% 9.1% 54 . 5% 
14.3% 12.9% 17.7% 12.1% 55.3% 



* Top number is the row percentage, bottom is the column percentage. 

For the poetry interpretation ranl^ comparisons the ohi-square ^•'as 30.99, again 
vrais a significance level of .0000. The results paralleled the other events in term? 
of percentage patterns. 

Wien considered individually ;>r collortively it b<N-om«;h, ;ii.»p\rcnt that there- is 
not a high degree of direct agreement between the forensics and lay Judge. Ho^^-'ever, 
it might be argued that as long as the two arc close in their ranldngs then there may 
be a case for suggesting some f om of agreement . Vx'hat then becomes necessary is ta 
generate a definition of a "closo" ranlving. If a performance is 'evaluated by two 
judges so that on]> one* rank ^.n^'in^^es the assessment i1 migh* be argued that these 
Judges are '•close" in their ranlvings. To meet this '.-riteria rarilts wore collapsed as 
^ -ndicated in Table 1 in the previous: section. The results are summarized in Table 7. 
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T.M3LE 7 
ADJACENT RANKINGS 

Forensics Judge Lay Judge Percentage of Occurrence __ 

Rank Adjacent Ranlt All Ex-ents Informative Persuasion Drama ^etQ- 

1st lst/2nd 64.1 66.7 69.2 69.2 45.9 

2nd lst/2nd/3ni 70.2 76.2 67.7 75.0 67 3 

3ni 2nd/3rxi/4th 59.7 76.2 6d.3 68.^ 44.3 

4th 3rd/4th/5th 72.4 76.2 73.4 82.1 63.9 

5th 4th/5th 70.6 63.4 81.1 ,7.1 o3.6 

Ave. % of occurren-e 67.4 71.7 73.8 74.0 5,.0 

This data does suggest that there is a degree of "closeness" between the rankings of 
forensics and lay Judges. When the results of ranldngs for all events is collapsed 
there utxs an average of 67.4 percent close agreement. However , if considered from 
the opposite vieu-point it might be argued that even with combined ranks there was 
still a high percentage of ranlis that were not "close". .\s Table 3 suggests, the 
range was from 25 percent to 43 percent of the rankings which fell outside- a "clos.>" 
similarity. 

TALLE 8 

nrniRRENCE OF W IDELY 
DRTOGENT RAN KS 

All Events 32.6% 

Informative Spealdng 28.3% 

Drama Interpretation 25.5% 

Persuasive Spealdng 26.2% 

Poetry 43.0% 

Depending on the perspective t^ien this collapsing of ranks may or ma> not sug- 
gest a similarity between forensics and lay judges. However, ^±cn considered in 
light of the earlier chi -square factoring the evidcncs? sooms to suggest a rej-iction 
of the second hypothesis. 
Di scuss i on /Concl us i onr 

The implications of this, st.udy are directly related to n uumb^^r of factors. 
Initially it must be noted that cv-n within the ranl«-s of forensics judges a wide 
range of ranlts is conr..on. This is something that oc^curs regularly during final 
:Rj(^-rounds of competition where multiple judges arc used. (As a side bar it should be 



pointed out that in final rounds the difference between performances is often so 
sisall that a judge may feel 1st through 5th ranked performances arc iiear identical, a 
situation which is not as common in preliminary rounds). Secondly, it can be argued 
that forensics is an art form and as such a wide spectrum of evaluation is likely. 

Yet it must also be noted that this art form is approached quite scientifically 
by performers, coaches, and judges. It is an art form with many rules, .^nd it is 
this set of rules that may be the real cause for the dissimilar rankings noted in 
this study. It seems that a question should be asked about why a lay audience would 
feel so different about a performance? Could it be that the standards and guidelines 
employod by the forensics community and reflected in the judging of performers have 
become more representative of a specialized environment than of society as a uiiole? 
And this leads to a most critical question: If this is truo is it bad? Is it desir- 
able that the forensic community's standards mirror a lay audience's criteria of uhtxt 
makes a performance "good"? Referring back to the opening analogy, a case can be 
made that, Jike the Olpipic figure skater, a forensics comix^tition represents the 
upper echelon of public communicalors and therefore should be judged by a set of 
standards that are above the norm* Such a position is both nuble an educationally 
sound. However, that same zest for the ideal may result ir: the acceptance of crite- 
ria tvhich may or may not be doing justice to the perfonner. For instance it is nut 
uncommon for a performance to be* judged in part on the numlx?r of times the material 
has been done in the past. Material that is "well traveled" is often assessed nega- 
tively by forf»nsics judges yel a la.\ audience i^'ould most likely not consider the 
frequency of performance as a major criteria. .\s anothci* 'ixamplo forensics judge may 
negatively evaluate a performer bc^oause they opt^n their script during the introduc- 
tion, yet this wouldn't bother a jay audience. These examples suggest that there may 
be a question concerning the appropriateness, and usefulness of some of the judging 
criteria used in this activity. 

It Wiixy well be that the distin(?tions created arc desirable, if so, then results 
such as those in this study are of little consequence. Or; the other hand if a major 
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aspect of forensics is to help student competitors reach a rarified level of comniuni- 
cation behavior v.iiich is readily adaptable to a general population and setting, then 
this study may indicate the need for further investigation into the standard used by 
forensic^ activists. 
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